ASH is a stateless Rust daemon that bridges Kaspersky Container Security (KCS) and downstream tooling. It receives inbound webhooks from KCS when a scan completes, resolves the scanned image to its scan record via the KCS REST API, fetches all four finding types (vulnerabilities, malware, misconfigurations, secrets), and POSTs a normalized JSON report to a configurable outbound URL.
KCS (scan complete)
│ POST /webhook {"image_name":…,"registry_name":…,"response_policy_name":…}
▼
┌─────────────────────────────────────────────────────────────────┐
│ ASH (Kubernetes pod, namespace kcs) │
│ │
│ 1. Resolve image → scan ID (registry list + sub-list APIs) │
│ 2. Fetch findings in parallel (4 scan-scoped endpoints) │
│ 3. Normalize → Report JSON (event_id, summary, findings[]) │
│ 4. POST report → KCS_OUTPUT_URL │
└─────────────────────────────────────────────────────────────────┘
│ POST {event_id, scan_id, summary, findings[]} (KCS_OUTPUT_FORMAT=default)
│ POST multipart hub.json (KCS_OUTPUT_FORMAT=appsechub)
▼
Downstream (SIEM, dashboard, pipeline, or AppSecHub)
- Prerequisites
- Configuration Reference
- Local Development
- Building the Container Image
- Kubernetes Deployment
- KCS Network Setup
- Output JSON Schema
- Troubleshooting
| Tool | Version | Purpose |
|---|---|---|
| Rust | ≥ 1.85 | Compilation (edition 2021, crates require rustc ≥ 1.86 via Cargo.lock) |
| Docker / nerdctl | any | Container image build |
| kubectl | ≥ 1.25 | Kubernetes deployment |
| kustomize | ≥ 5 | Overlay-based configuration (bundled in kubectl ≥ 1.14) |
Runtime dependencies: none. ASH compiles to a single binary; the container image is based on debian:bookworm-slim for CA certificate support.
All configuration is supplied via environment variables or CLI flags. CLI flags override environment variables. In Kubernetes, non-sensitive values come from a ConfigMap and the API key from a Secret.
| Env var | CLI flag | Default | Required | Description |
|---|---|---|---|---|
KCS_BASE_URL |
--base-url |
— | yes | KCS API base URL, e.g. https://kcs.example.com (no trailing /api) |
KCS_API_KEY |
--api-key |
— | yes | KCS static API key (sent as Tron-Token header). Never logged. |
KCS_OUTPUT_URL |
--output-url |
— | yes | URL to POST findings JSON to, e.g. https://siem.internal/ingest |
KCS_LISTEN_PORT |
--listen-port |
8080 |
no | TCP port for inbound webhook server |
KCS_WEBHOOK_PATH |
--webhook-path |
/webhook |
no | HTTP path for the inbound webhook endpoint |
KCS_CA_CERT_PATH |
--ca-cert |
(unset) | no | Path to PEM CA certificate; trusted for both KCS API and output URL TLS |
KCS_VERIFY_SSL |
--verify-ssl |
true |
no | Set to false to disable TLS certificate verification (test environments only) |
KCS_MAX_CONCURRENT_EVENTS |
--max-concurrent |
16 |
no | Semaphore cap on in-flight webhook processing tasks; 503 returned when full |
KCS_HTTP_TIMEOUT_SECS |
--http-timeout |
30 |
no | Per-attempt timeout in seconds for all outbound HTTP |
KCS_READINESS_STALE_SECS |
--readiness-stale |
600 |
no | Max age (seconds) of last successful KCS API call for /ready to return 200 |
KCS_RESOLVE_MAX_ATTEMPTS |
--resolve-max-attempts |
6 |
no | Attempts to find the scan in the registry sub-list before giving up (absorbs KCS scan-commit lag) |
KCS_RESOLVE_BACKOFF_BASE_SECS |
--resolve-backoff-base-secs |
1 |
no | Base backoff (seconds) between sub-list attempts; doubles each attempt (1, 2, 4, …), capped at 30s per wait |
KCS_OUTPUT_FORMAT |
--output-format |
default |
no | Output format for delivered reports: default (normalized JSON) or appsechub (AppSecHub v1.0.1 schema via multipart form-data). See AppSecHub Output. |
KCS_HUB_TOKEN |
--hub-token |
— | required when appsechub |
AppSecHub JWT token, sent as X-Authorization: token <value>. Never logged. |
KCS_HUB_APP_ID |
--hub-app-id |
— | required when appsechub |
AppSecHub application ID (positive integer), sent in the multipart metadata part. |
KCS_HUB_SOURCE_URL |
--hub-source-url |
(unset) | required when appsechub |
Docker registry base URL registered in AppSecHub as the scanned artifact (e.g. nexus.example.com/kaspersky). ASH appends the image basename from the KCS webhook to form a full reference: nexus.example.com/kaspersky/event-broker:v2.4.0. Must match the artifact URL registered in the AppSecHub application's Разработка (Development) tab. |
KCS_WEBHOOK_DEBUG |
--webhook-debug |
false |
no | Log the raw body of every inbound webhook request at INFO level. Useful when diagnosing unexpected KCS payload formats; leave disabled in production to avoid noise. |
KCS_BASE_URL and KCS_OUTPUT_URL have no defaults because they differ per environment. The daemon exits with code 1 at startup if either is absent or empty.
KCS fires the scan-completion webhook the instant a scan finishes, often before the scan record is committed to the registry sub-list that ASH queries. ASH therefore retries the sub-list lookup with exponential backoff. With the defaults (KCS_RESOLVE_MAX_ATTEMPTS=6, KCS_RESOLVE_BACKOFF_BASE_SECS=1) it polls at 0s, 1s, 2s, 4s, 8s, 16s — a ~31-second window. Under bulk container scans the commit lag is larger than a single scan's, so if you see artifact not found in sub-list after N attempts errors, raise KCS_RESOLVE_MAX_ATTEMPTS.
The processing semaphore permit is held for the whole resolve→fetch→deliver pipeline, including these backoff waits, so a longer retry window means each lagging event occupies a permit longer. If a dense webhook burst then hits the KCS_MAX_CONCURRENT_EVENTS cap (503s), raise that limit too.
The KCS API key is a static string passed as Tron-Token: <key> on every KCS API request. Obtain it from KCS Settings → API Access. Typical format: kcs_<alphanumeric>.
The key is:
- Never written to logs at any level
- Not accepted as a positional CLI argument (would be visible in
ps) - Validated against
/api/v1/healthzat startup; the daemon exits if rejected
KCS and the output URL often use certificates signed by an internal CA not in the system trust store.
Preferred — mount the CA cert:
# Find the CA that signed the KCS ingress certificate
openssl s_client -connect kcs.example.com:443 2>/dev/null | openssl x509 -noout -issuer
# Extract the CA cert (ask your KCS admin, or find it in your deployment files)
# Store it in Kubernetes as a ConfigMap:
kubectl create configmap ash-ca-cert -n kcs --from-file=ca.crt=/path/to/ca.crtSet KCS_CA_CERT_PATH=/etc/ash/ca.crt and mount the ConfigMap at /etc/ash in the Deployment. The same CA is trusted for both the KCS API and the output URL.
Fallback — disable TLS verification:
KCS_VERIFY_SSL=false
A warning is printed to stderr at startup. Use only in test environments when the CA is not available.
Important: The CA cert for the KCS ingress (
Generic-KCS-CAin the reference environment) is not the same as thecert-caKubernetes Secret (which is the internal KCS service-mesh CA). You need the CA that signed the ingress TLS certificate.
# Clone and enter the project
git clone <repo>
cd ash
# Run tests (unit + integration + doctests, 28 total)
cargo test
# Build debug binary
cargo build
# Build release binary
cargo build --release
# Binary at: target/release/ash
# Run locally (requires a KCS instance)
KCS_BASE_URL=https://kcs.example.com \
KCS_API_KEY=kcs_yourkey \
KCS_OUTPUT_URL=https://webhook.site/your-id \
KCS_VERIFY_SSL=false \
./target/release/ashcargo test # all tests
cargo test resolver # unit tests for image name parser
cargo test -- --nocapture # show println! outputTests use httpmock for integration scenarios and do not require a live KCS instance.
ASH emits structured JSON logs to stdout (one JSON object per line):
{"timestamp":"2026-05-28T19:43:35Z","level":"INFO","fields":{"message":"KCS healthz OK — starting server","version":"2.4.0"},"target":"ash"}
{"timestamp":"2026-05-28T19:43:35Z","level":"INFO","fields":{"message":"listening","addr":"0.0.0.0:8080","webhook":"/webhook"},"target":"ash"}
{"timestamp":"2026-05-28T19:44:30Z","level":"INFO","fields":{"message":"delivered report","event_id":"…","scan_id":"…","total":68,"vulnerabilities":29,"malware":2,"misconfigurations":36,"secrets":1},"target":"ash::sender"}The Dockerfile is a two-stage build: Rust compiler image → debian:bookworm-slim runtime.
docker build -t ash:v0.1.2 .
docker tag ash:v0.1.2 registry.example.com/ash/ash:v0.1.2
docker push registry.example.com/ash/ash:v0.1.2Kubernetes reads images from the root containerd k8s.io namespace, and the
deployment uses imagePullPolicy: Never, so the image must exist there — no
registry involved.
If a rootful buildkitd is reachable, build straight into that namespace:
sudo nerdctl --namespace k8s.io build -t ash:v0.1.2 .If only a rootless buildkitd is running (common on these hosts — rootful
buildkit is often not reachable), build rootless and then load the image into
the root k8s.io namespace:
# 1. Build with the rootless builder
nerdctl build -t ash:v0.1.2 .
# 2. Move the image into the namespace Kubernetes uses
nerdctl save docker.io/library/ash:v0.1.2 | sudo nerdctl --namespace k8s.io load
# 3. Verify it is visible to Kubernetes
sudo nerdctl --namespace k8s.io images | grep ashThe Cargo.lock is committed and is authoritative. The Dockerfile pins the
builder image to rust:1.95-bookworm for reproducible builds; the pinned crate
versions (icu 2.2 family) require rustc ≥ 1.86, so do not downgrade below that.
The deploy/ directory contains Kustomize manifests:
deploy/
├── base/ # shared across all environments
│ ├── kustomization.yaml
│ ├── secret.yaml # placeholder — never commit real keys
│ ├── configmap.yaml # non-sensitive env vars with placeholders
│ ├── deployment.yaml
│ └── service.yaml
└── overlays/
├── test/ # test environment values
│ └── kustomization.yaml
└── production/ # production placeholders
└── kustomization.yaml
kubectl create secret generic ash-secret \
--namespace kcs \
--from-literal=KCS_API_KEY=kcs_yourkeyDo not apply deploy/base/secret.yaml with a real key — it contains only a placeholder and is safe to commit.
kubectl create configmap ash-ca-cert \
--namespace kcs \
--from-file=ca.crt=/path/to/generic-kcs-ca.crt# Test environment
kubectl apply -k deploy/overlays/test
# Production (after editing deploy/overlays/production/kustomization.yaml)
kubectl apply -k deploy/overlays/production| Value | Where | Example |
|---|---|---|
KCS_BASE_URL |
ConfigMap overlay patch | https://kcs.prod.example.com |
KCS_OUTPUT_URL |
ConfigMap overlay patch | https://siem.prod.example.com/ingest |
| Container image | Kustomize images: transformer |
registry.prod.example.com/ash/ash:v0.1.2 |
Single replica with Recreate strategy: Two simultaneous pods would both receive webhook events from KCS, causing duplicate reports. strategy: Recreate accepts a brief downtime during rollout in exchange for unambiguous delivery.
Control-plane scheduling: If deploying to a cluster where worker nodes are not reachable for image distribution, force the pod to run where the image was built using a toleration and nodeSelector:
# Add to spec.template.spec in deployment.yaml
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
nodeSelector:
kubernetes.io/hostname: control.demo.labAnd set imagePullPolicy: Never in the container spec to use a locally-built image without a registry.
| Endpoint | Type | Returns 200 when |
|---|---|---|
GET /health |
Liveness | axum server is running |
GET /ready |
Readiness | startup healthz passed AND last KCS API success ≤ KCS_READINESS_STALE_SECS ago |
The readiness probe ensures the pod is only added to Service endpoints after the API key is validated. A revoked key causes /ready to flip to 503 within KCS_READINESS_STALE_SECS (default 10 min), which removes the pod from load balancing while keeping it alive for inspection.
# The runtime image (debian:bookworm-slim) ships no curl/wget and no shell
# tooling for exec probes, so check the endpoints from outside via port-forward:
kubectl port-forward -n kcs deploy/ash 8080:8080 &
curl -s http://localhost:8080/health # {"status":"ok"}
curl -s http://localhost:8080/ready # {"status":"ready"}http://ash.kcs.svc.cluster.local:8080/webhook
This is the internal Kubernetes DNS name. KCS and ASH run in the same namespace (kcs) and communicate over the cluster network — no ingress, no internet exposure required.
KCS UI path: Settings → Response Policies → [your policy] → Webhook URL
For ASH to receive webhooks, configure a KCS Response Policy:
- Create a policy with action Webhook
- Set the webhook URL to
http://ash.kcs.svc.cluster.local:8080/webhook - Bind the policy to specific images or registries (not a global/unscoped policy)
If
image_namearrives as the literal string"ImageName": the policy is not bound to a specific image. KCS sends"ImageName"as a placeholder when the policy is triggered globally without image context. Scope the policy to the specific registry and image.
KCS sends the following JSON body when a policy triggers:
{
"response_policy_name": "WH_VULN",
"image_name": "registry.example.com/team/app:v1.0",
"registry_name": "MyRegistry"
}ASH accepts any response_policy_name value — it does not filter by policy name.
Each delivered report is a single JSON object:
{
"event_id": "f4f319ff-fa86-4c51-8d05-73f14bab851b",
"exported_at": "2026-05-28T19:46:13.255Z",
"image_name": "jfrog.tronsec.ru:443/demo-tron/bad:bad-project-test",
"scan_id": "8659ab75-b395-4166-965e-4b03bec52d6e",
"summary": {
"vulnerabilities": 29,
"malware": 2,
"misconfigurations": 36,
"secrets": 1,
"total": 68
},
"findings": [
{
"type": "vulnerability",
"id": "…",
"cve_id": "CVE-2022-37434",
"severity": "critical",
"package": "zlib 1.2.12-r1",
"fixed_version": "1.2.12-r2",
"image": "jfrog.tronsec.ru:443/demo-tron/bad:bad-project-test"
},
{
"type": "malware",
"id": "…",
"name": "Virware:EICAR-Test-File",
"severity": "high",
"image": "…",
"path": "malware/eicar-2.com.txt",
"file_hash_md5": "44d88612fea8a8f36de82e1278abb02f",
"file_hash_sha256": "275a021b…",
"scan_id": "8659ab75-…"
},
{
"type": "misconfiguration",
"id": "…",
"title": "API Gateway domain name uses outdated SSL/TLS protocols.",
"severity": "high",
"image": "…",
"scan_id": "8659ab75-…"
},
{
"type": "secret",
"id": "…",
"title": "AWS Secret Access Key",
"severity": "critical",
"image": "…",
"scan_id": "8659ab75-…"
}
]
}Deduplication: use event_id (per delivery) or scan_id (per scan) for downstream deduplication. event_id is a fresh UUIDv4 per webhook event. scan_id is stable for a given scan result and the same across retries of the same event.
| Response | Action |
|---|---|
| 2xx | Success — event complete |
| 5xx, 408, 429, network error | Retry: wait 1 s, retry; wait 2 s, retry; then drop |
| Other 4xx (400, 401, 403, 404…) | Permanent failure — drop immediately, no retry |
When KCS_OUTPUT_FORMAT=appsechub, ASH delivers scan results to an AppSecHub instance using the v1.0.1 schema.
Instead of the default normalized JSON body, the report is delivered as a multipart/form-data POST to KCS_OUTPUT_URL with:
- Header
X-Authorization: token <KCS_HUB_TOKEN> - Part
json:{"application": {"appId": <KCS_HUB_APP_ID>}}(metadata) - Part
reportFile: AppSecHub v1.0.1 JSON report (hub.json)
The hub report structure:
- One deduplicated rule per unique CVE ID
- One deduplicated location per unique
package@version(derived by splittingpackageNameon its last space) - One finding per unique (CVE, component) pair, with a deterministic MD5 id
AppSecHub mode covers vulnerability findings only. Malware, misconfiguration, and secret findings are not included in the hub report — the AppSecHub v1.0.1 schema only defines the sca_s type for container image CVEs. A startup info log confirms this exclusion.
To include all finding types, use KCS_OUTPUT_FORMAT=default.
AppSecHub's onboarding integration requires the source[].url field in the submitted report to match (as a prefix) a Docker artifact registered in the target application's Разработка (Development) tab. ASH constructs the URL as:
<KCS_HUB_SOURCE_URL>/<image-basename-from-webhook>
For example, with KCS_HUB_SOURCE_URL=nexus.test.swordfishsecurity.com/kaspersky and a KCS webhook for repo.kcs.kaspersky.com/images/services/event-broker:v2.4.0, the report source URL becomes:
nexus.test.swordfishsecurity.com/kaspersky/event-broker:v2.4.0
How to find the right value:
- In AppSecHub, open the target application → Разработка (Development)
- Find the Docker artifact whose repository URL covers the images being scanned
- Use that artifact's URL as
KCS_HUB_SOURCE_URL
Test environment value: nexus.test.swordfishsecurity.com/kaspersky
(This is the Docker registry registered in the demo AppSecHub instance for application #171 "Kaspersky - KCS".)
If this parameter is empty, ASH uses the raw KCS image URL verbatim — which will fail onboarding unless that exact URL is registered as an artifact in AppSecHub.
# Required
KCS_OUTPUT_FORMAT=appsechub
KCS_OUTPUT_URL=https://demo.appsec-hub.ru/hub/rest/integration/report
KCS_HUB_TOKEN=eyJhbGciOiJIUzUxMiJ9... # JWT from AppSecHub Settings → Integrations
KCS_HUB_APP_ID=171 # AppSecHub application ID (from URL: #/appprofile/171/...)
KCS_HUB_SOURCE_URL=nexus.test.swordfishsecurity.com/kaspersky # registry artifact URL from Development tabIn Kubernetes (patch the existing ConfigMap):
kubectl patch configmap ash-config -n kcs --type merge -p '{
"data": {
"KCS_OUTPUT_FORMAT": "appsechub",
"KCS_OUTPUT_URL": "https://demo.appsec-hub.ru/hub/rest/integration/report",
"KCS_HUB_APP_ID": "171",
"KCS_HUB_SOURCE_URL": "nexus.test.swordfishsecurity.com/kaspersky"
}
}'
# KCS_HUB_TOKEN goes in the Secret (not the ConfigMap):
kubectl patch secret ash-secret -n kcs --type merge -p \
'{"stringData":{"KCS_HUB_TOKEN":"eyJhbGciOiJIUzUxMiJ9..."}}'
kubectl rollout restart deployment/ash -n kcs- Deploy with
KCS_OUTPUT_FORMAT=default(or unset) — no behavioral change. - Set all five AppSecHub variables (
KCS_OUTPUT_FORMAT,KCS_OUTPUT_URL,KCS_HUB_TOKEN,KCS_HUB_APP_ID,KCS_HUB_SOURCE_URL) and restart. - Trigger a KCS rescan and verify the onboarding task in AppSecHub Журнал задач → Онбординг shows a non-СЛОМАН status.
- Check Уязвимости (Issues) in the target application — new findings should appear within seconds of the task completing.
- Rollback: unset
KCS_OUTPUT_FORMATor set it back todefault— instant, no restart required.
Cause: KCS fires the webhook before the new scan record is committed to the sub-list API (race condition). ASH retries the sub-list lookup with exponential backoff — by default 6 attempts over ~31s (KCS_RESOLVE_MAX_ATTEMPTS / KCS_RESOLVE_BACKOFF_BASE_SECS, see Scan resolution & commit lag). This commonly appears during bulk container scans, where the commit lag exceeds a single scan's. If it persists, either the lag is still longer than the retry window (raise KCS_RESOLVE_MAX_ATTEMPTS), the image was deleted, or the registry_name in the webhook doesn't match a registered registry.
Check — confirm the artifact is actually present and what ASH expects (<short-name>:<tag>):
# 1. Find the registry entry id for the repository
RID=$(curl -sk -H "Tron-Token: $KCS_API_KEY" \
"$KCS_BASE_URL/api/v1/images/registry?limit=1000" \
| jq -r '.items[] | select(.name=="<repository>") | .id')
# 2. List the artifactName values in its sub-list
curl -sk -H "Tron-Token: $KCS_API_KEY" \
"$KCS_BASE_URL/api/v1/images/registry/$RID/sub-list?limit=100" \
| jq -r '.items[].artifactName'If the artifact is listed but ASH still failed, the scan was committed after ASH gave up — increase the retry window.
Cause: KCS Response Policy is not scoped to a specific image. The literal string "ImageName" is KCS's unsubstituted template placeholder.
Fix: In the KCS UI, edit the Response Policy and bind it to a specific registry image rather than leaving the image scope empty.
The file at KCS_CA_CERT_PATH does not exist or is not readable. Verify the ConfigMap is mounted correctly:
kubectl describe pod -n kcs -l app=ash | grep -A5 "Mounts"
kubectl exec -n kcs deploy/ash -- ls -la /etc/ash/kubectl logs -n kcs -l app=ash --previous # logs from the crashed container
kubectl describe pod -n kcs -l app=ash # events sectionCommon causes:
KCS_BASE_URL/KCS_API_KEY/KCS_OUTPUT_URLnot set (exits code 1 immediately)KCS_CA_CERT_PATHset but file not mounted (exits code 1)- Startup healthz fails — wrong
KCS_BASE_URLor invalid API key runAsNonRoot: truewith a named (non-numeric) user — addrunAsUser: 1000to the containersecurityContext
The background healthz ping has not succeeded within KCS_READINESS_STALE_SECS. Check:
- KCS API reachability from the pod
- API key validity
KCS_CA_CERT_PATHis correct and the CA cert matches the KCS ingress certificate
The KCS ingress certificate is signed by a separate CA from the internal KCS service-mesh CA (cert-ca secret). To find the correct CA:
# See what CA signed the KCS ingress cert
echo | openssl s_client -connect kcs.example.com:443 2>/dev/null | openssl x509 -noout -issuer
# Ask your KCS admin for the CA cert file, or look in the KCS deployment directory
# In the reference environment: ~/kcs/certs/ca_public.crtVerify it chains correctly before mounting:
openssl verify -CAfile ca_public.crt kcs-ingress.crtASH always logs "delivered report" on HTTP 200, but AppSecHub processes the submission asynchronously. A task can still fail after the HTTP 200 is returned.
Check the task log: AppSecHub → application → Журнал задач → Онбординг tab → click the warning icon on the broken task.
Common error messages and fixes:
| Error | Cause | Fix |
|---|---|---|
Application id = N is not linked with suitable repositories |
The application has no Docker artifact registered that matches the report source URL. | In Разработка (Development), add the Docker artifact whose registry URL matches KCS_HUB_SOURCE_URL. |
Unable to parse an artifact url: url not matches with template |
KCS_HUB_SOURCE_URL is set to a registry base path (e.g. nexus.example.com/ns) without an image name and tag. ASH appended the KCS image basename automatically — verify the resulting URL looks like registry/image:tag. |
Ensure KCS_HUB_SOURCE_URL is a valid registry-namespace prefix (no trailing slash, no image name). ASH will append the basename. |
permanent delivery failure: HTTP 401 |
KCS_HUB_TOKEN is expired or invalid. |
Re-issue the JWT in AppSecHub Settings → Integrations and update the Kubernetes Secret. |
Reports are POSTed to KCS_OUTPUT_URL. For testing, use webhooktest.net:
- Receive URL:
https://webhooktest.net/webhook/<your-bucket-id>(POST endpoint) - View URL:
https://webhooktest.net/bucket/<your-bucket-id>(browser)